We propose a meta-learning method for learning from multiple noisy annotators. In many applications such as crowdsourcing services, labels for supervised learning are given by multiple annotators. Since the annotators have different skills or biases, given labels can be noisy. To learn accurate classifiers, existing methods require many noisy annotated data. However, sufficient data might be unavailable in practice. To overcome the lack of data, the proposed method uses labeled data obtained in different but related tasks. The proposed method embeds each example in tasks to a latent space by using a neural network and constructs a probabilistic model for learning a task-specific classifier while estimating annotators' abilities on the latent space. This neural network is meta-learned to improve the expected test classification performance when the classifier is adapted to a given small amount of annotated data. This classifier adaptation is performed by maximizing the posterior probability via the expectation-maximization (EM) algorithm. Since each step in the EM algorithm is easily computed as a closed-form and is differentiable, the proposed method can efficiently backpropagate the loss through the EM algorithm to meta-learn the neural network. We show the effectiveness of our method with real-world datasets with synthetic noise and real-world crowdsourcing datasets.

Two of the reviewers increased their score after reading the rebuttal. All three reviewers now provide accepting scores. The reviewers particularly appreciated the authors response. In particular the additional experiment on mini-imagenet as it re-confirms the original idea and gives consistent results as those obtained with simpler datasets. The idea of borrowing meta-learning ideas to tackle the continual learning problem is interesting and the empirical results sufficient.

However, the rationale of OML in Eqn.3 is not sufficient to support the "correlated sequences". Even though consulting with Appendix B, the rationale is weak to persuade. Do you assume that the k-step online update in Eqn.3 can give optimal loss for RLN although k is smaller than the length of each session? And, what do you mean by "finding a model initialization and learning a fixed representation such that starting from the learned representation it has xyz properties (Appendix L379-380)"? - I strongly recommend the authors to move Algorithm 1 from Appendix to the paper after polishing Section 2 & 3. * Related works Missing related works make it hard to assess the novelty and significance of the proposed method. If it is possible, please do report the controlled experiments to compare state-of-the-art.

The reviews had two major concerns: lack of a benchmarking on a complex dataset, and unclear writing. To address these two major issues we: 1- Rewrote experiments section with improved terminology to make the paper more clear. Previously we were using the term Pretraining to refer to both a baseline and the meta-training stage. As the reviewers pointed out, this was confusing. We have replaced one of the usages with'meta-training.'

For better clustering performance, appropriate representations are critical. Although many neural network-based metric learning methods have been proposed, they do not directly train neural networks to improve clustering performance. We propose a meta-learning method that train neural networks for obtaining representations such that clustering performance improves when the representations are clustered by the variational Bayesian (VB) inference with an infinite Gaussian mixture model. The proposed method can cluster unseen unlabeled data using knowledge meta-learned with labeled data that are different from the unlabeled data. For the objective function, we propose a continuous approximation of the adjusted Rand index (ARI), by which we can evaluate the clustering performance from soft clustering assignments. Since the approximated ARI and the VB inference procedure are differentiable, we can backpropagate the objective function through the VB inference procedure to train the neural networks. With experiments using text and image data sets, we demonstrate that our proposed method has a higher adjusted Rand index than existing methods do.

The reviews had two major concerns: lack of a benchmarking on a complex dataset, and unclear writing. To address these two major issues we: 1- Rewrote experiments section with improved terminology to make the paper more clear. Previously we were using the term Pretraining to refer to both a baseline and the meta-training stage. As the reviewers pointed out, this was confusing. We have replaced one of the usages with'meta-training.' We have also changed evaluation to meta-testing.